Regression & Statistical Modeling: The Line, and What It Means

Why do tall parents tend to have tall children, but children of very tall parents are on average a bit shorter than their parents? Why does a restaurant that wins an award one year tend to do slightly worse the next? Why does a model that fits last year's data perfectly predict this year's badly? Each is a regression story, and each is told wrongly all the time.

Regression answers a plain question: how does one number change, on average, as another changes? The answer is a line, and a line is easy to draw. What takes a course is learning to read it: what the slope means when other things are also moving, how much the slope would wobble with a different sample, when a better-fitting curve is a worse predictor, and how to model a yes-or-no outcome.

The history is short and oddly argumentative. Adrien-Marie Legendre published the method of least squares in 1805. Carl Friedrich Gauss claimed in 1809 that he had been using it since 1795, which began a priority dispute. Francis Galton coined the word "regression" in a paper of 1886 about the heights of parents and children. The word stuck to the method, though it describes only the first of its uses.

Least squares: the line that makes the squares smallest

Given a cloud of points, which line is "best"? Any line leaves a vertical gap at each point, called a residual: the observed value minus the line's prediction. A good line has small gaps. The question is how to add them up. Positive and negative gaps cancel, so you square them first. The line that makes the sum of squared residuals as small as possible is the least-squares line.

Squaring has a physical picture: each residual is the side of a square, and you are trying to make the total area of those squares small. It also has a mathematical reason: squares make the problem a smooth bowl with exactly one lowest point, so a formula (or one pass of linear algebra) finds it. Large misses are punished heavily, which is both the method's strength and its weakness, since one wild point can drag the whole line.

Try it. Your line starts in a poor place. Move the sliders to shrink the squares, then compare your total with the true minimum. Drag the points (or tap where you want one) and watch the best line move; pull one point far from the rest to see how much leverage it has.

Without JavaScript: for the starting nine points the least-squares line is about y = 1.4 + 0.67x with an R² of 0.95. A poor guess such as y = 4 + 0.3x has a sum of squares about ten times larger.

R² and the residuals: what the line explained and what it missed

How good is the fit? One summary is R², the share of the outcome's variation that the line accounts for. If you predicted every point with the plain average, your squared misses would total some amount (the total variation). The line's squared misses are what is left over. R² is the fraction of the total that the line removed.

A single R² can hide a poor model, which is the lesson of Anscombe's quartet in Data Wrangling & Exploratory Analysis (36/31). The remedy is to look at the residuals. If the line is right, residuals scatter randomly around zero with the same spread everywhere. Patterns mean the model is missing something: a bow means the relationship is curved, a funnel means the noise grows with the input, and one far-off point can tilt everything. Pick a pattern below and compare the R² with the residual plot.

Without JavaScript: the curved pattern still has an R² of about 0.86 although its residuals form a clear bow; the funnel has residuals about three times as spread out on the right as on the left; removing the single outlier changes the fitted slope noticeably and lowers the R².

R² is also easy to inflate: adding any input, even a useless one, never lowers it on the data you fitted. That is why serious reports pair it with an out-of-sample check, which is the topic of the overfitting section below.

Reading a coefficient: "holding the others fixed"

With one input, the slope says how much the outcome changes on average when the input rises by one unit. With several inputs, each coefficient is read with a phrase attached: the change in the outcome per unit of this input, holding the other inputs fixed. The computer finds all coefficients at once so that, in effect, each input is compared only among cases that match on the others.

That phrase is where regression earns its keep. Take children aged 5 to 12. Those with bigger feet read better. But no one thinks feet teach reading. Both grow with age. In the simulation below the true effect of shoe size on reading is a number you choose, and the default is zero. Compare the simple slope with the slope after age is included.

Without JavaScript: with a true effect of zero, the simple slope of reading on shoe length is about 3.6 points per cm and looks strongly "significant"; once age is in the model the shoe coefficient falls to about 0.6, with a confidence interval that runs from below zero to above 1.

Two cautions. A coefficient can only be adjusted for inputs you measured, so an unmeasured confounder survives. And "holding fixed" is a statistical statement about comparing similar cases, not proof that changing one thing in the world would change the outcome. That jump needs a design, not just a regression; Experiments & Causal Inference (36/34) is the course for it. A related trap, Simpson's paradox, appeared in Data Wrangling & Exploratory Analysis (36/31).

The slope wobbles: standard errors and confidence intervals

A fitted slope is an estimate from one sample. Collect another sample from the same world and you would get a slightly different line. The standard error says how much, typically. For a single input it is the typical size of a residual divided by the square root of the total spread of x:

The dependence on data is a square root: four times as many observations halves the standard error, and ten times as many cuts it to about a third.

A 95% confidence interval is the slope plus and minus a multiple of the standard error (about 1.96 for large samples, larger for small ones). The meaning is easy to garble. It is a statement about the procedure: if you repeated the whole study many times, about 95% of the intervals built this way would contain the true slope. It is not a 95% probability that this particular interval holds it, in the strict frequentist reading. Bayesian methods answer a different question, with different assumptions, and statisticians still argue about which is better to teach.

See the wobble yourself. The world below has a true slope of 0.5. Choose a sample size, draw 30 samples, and look at the bundle of fitted lines. Then watch what 200 repeated samples say about the standard error and about how often the interval catches the truth.

Without JavaScript: at n = 20 the standard error of the slope is about 0.16 and the 95% intervals catch the true slope in roughly 93 to 95% of repeated samples; at n = 200 the standard error is about 0.05. Ten times the data cuts the wobble to about a third, the square root of ten.

Overfitting: a flexible model learns the noise

A straight line is rigid. A curve with many bends is flexible, and with enough bends it can pass through every training point. That sounds like success until you ask how it does on new data. A model that has memorised the noise of one sample has learned nothing portable.

The standard defence is to hold some data back: fit on a training set, then measure the error on a test set the model never saw. Training error always falls as flexibility rises. Test error falls, then climbs. The bottom of that U is where the model has learned the signal and not yet the noise. The tension is the bias–variance trade-off: a rigid model is wrong in a steady way (bias), and a flexible one changes wildly with the sample (variance). Machine Learning (34/32) takes it much further.

Without JavaScript: for the starting sample, the training error falls steadily from 0.33 at degree 1 to 0.06 at degree 9, while the test error is lowest near degree 3 (about 0.41) and rises above 2 by degree 8.

Logistic regression: when the answer is yes or no

A straight line is a poor model of an outcome that is 0 or 1 (passed or failed, clicked or did not, survived or not). The line goes below 0 and above 1, and "0.7 of a pass" means nothing. Logistic regression fixes this by modelling the log-odds of the outcome as a straight line, then converting back to a probability with an S-shaped curve. The odds of an event are p ÷ (1 − p): a probability of 0.75 is odds of 3 to 1. David Cox's 1958 paper on the regression analysis of binary sequences is a standard reference for the method in statistics.

The coefficients are found by maximum likelihood rather than least squares: choose the curve under which the observed passes and fails were most probable. Slide the two parameters below to raise the likelihood, then let the page find the maximum.

Without JavaScript: for the invented data the best fit is near b₀ = −4.1 and b₁ = 1.16, an odds ratio of about 3.2 per hour, with a 50% chance of passing at about 3.6 hours of study.

Predict or explain: two jobs for one tool

Regression is used for two different purposes, and the standards differ. For prediction, the only question is whether the model does well on new cases; whether the coefficients mean anything is irrelevant. For explanation, the coefficients are the point, and you care about confounders, measurement error and whether the model is the right shape. In 2001 the statistician Leo Breiman described these as two cultures: one starting from an assumed data-generating model, the other from algorithms judged by predictive accuracy. His essay set off a debate that still runs.

One more idea explains the opening puzzles. Galton found that extremely tall parents had children who were tall but, on average, less extreme. This is regression to the mean. Where two measurements are imperfectly related, a case chosen for being extreme on the first is, on average, closer to the middle on the second, because part of what made it extreme was luck. Award-winning restaurants, record-breaking seasons and top-scoring students all drift back. It is not a force; it is selection plus noise, and many "cures" and "slumps" are nothing more.

What a graduate just sees

• The best line is the one that makes the squared vertical misses smallest, and one wild point can drag it.
• R² says how much of the variation the line removed, and it never says whether the line was the right shape.
• Residual plots are the first diagnostic: bows, funnels and lone outliers show up there.
• A coefficient means "per unit, holding the others fixed", and only the others you measured.
• Every estimate wobbles; the standard error shrinks with the square root of the sample size.
• More flexibility always improves the fit to the data in hand and often worsens the next prediction.
• For yes-or-no outcomes, model log-odds with a line and read coefficients as odds ratios.
• Extreme cases drift back toward the middle, with no cause needed.

Where it shows up next

• Experiments & Causal Inference (36/34): when a coefficient may be read as a cause, and what a randomised experiment, or an instrument, buys you.
• Econometrics (34/44): regression with panel data, instruments and standard errors that survive real-world messiness.
• Machine Learning (34/32) and Deep Learning (36/35): regularisation, cross-validation and neural networks are regression made flexible, with the overfitting problem made central.
• Data Ethics, Privacy & Fairness (36/37): a model that predicts well can still encode and amplify a pattern in the data; fitting it does not make it fair.
• In the world: insurance pricing, clinical risk scores, A/B-test analysis, sports analytics, credit scoring and every "the data show that" claim in the news.

If you want to go deeper

• 📘 Textbook. Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani, An Introduction to Statistical Learning (2nd edition, Springer, 2021; a Python edition followed in 2023), free from statlearning.com; or Andrew Gelman, Jennifer Hill and Aki Vehtari, Regression and Other Stories (Cambridge University Press, 2020), which is strong on reading coefficients honestly.
• 📘 Gentler. David Spiegelhalter, The Art of Statistics (2019), with a lively chapter on regression and regression to the mean; and Charles Wheelan, Naked Statistics (2013).
• 🌐 Free text. The two textbooks above are the standard routes. Computational and Inferential Thinking at inferentialthinking.com (Berkeley Data 8) has a chapter on regression built from first principles.
• 🌐 Free course. MIT OpenCourseWare 18.650 Statistics for Applications (Fall 2016, Philippe Rigollet), with lecture videos on linear and logistic regression; and 15.071 The Analytics Edge (Spring 2017), whose Unit 2 is linear regression.